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(57) Abstract 

A method of manipulating a complex waveform by considering the harmonic and partial frequencies as moving targets over time in 
both amplitude and frequency and adjusting the moving targets by moving modifiers in bom amplitude and frequency. The manipulation of 
harmonic frequencies and the synthesis of harmonic frequencies are based on the harmonic rank. The modifiers move with the movement 
of the frequencies based on rank. Harmonic transformation modifies, by rank, the waveform from one source to a waveform of a second 
or target source. Harmonics and other partials accentuation identifies each of (he frequencies and its relationship to adjacent frequencies as 
well as fixed or moving thresholds and make the appropriate adjustment Interpolation is also disclosed as well as models which imitate 
natural harmonics. 
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METHOD OF MODIFYING HARMONIC CONTENT OP A COMPLEX WAVEFORM 

CROSS-REFERENCE 

This application is related to and claims the 

benefit of Provisional Patent Application Serial No. 

60/106,150 filed October 29 , 1998 which is 

5 incorporated herein by reference, 

GLOSSARY, BACKGROUND, AND SUMMARY OF TKF INVM££gg 
The present invention relates generally to 
audio signal processing and waveform processing , and 
0 the modification of harmonic content of periodic 

audio signals and more specifically to methods for 
dynamically altering the harmonic content of such 
signals for the purpose of changing their sound or 
perception of their sound. 
5 Many terms used in this patent are collected 

and defined in this section. 

Among the many kinds of sounds which 
continually bombard the human ear, one is 
distinguished by its character of lasting long 
> enough and being steady enough for the ear to 

ascribe to it characteristics of amplitude, timbre, 
and pitch. This kind of sound is called a tone. 

The quality of the tone, or timbre, is the 
characteristic which allows it to be distinguished 
from other tones of the same frequency and loudness 
or amplitude. In less technical terms, this aspect 
gives a musical instrument its recognizable 
personality or character, which is due in large part 
to its harmonic content over time. 

Some musical instruments produce steady tones 
that can remain unchanged in character for at least 
a few seconds, long enough for several hundred 
cycles to take place. Such tones are said to be 
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periodic. 

Most sound sources, including musical 
instruments, produce complex waveforms that are 
mixtures of sine waves of various amplitudes and 
frequencies. The individual sine waves contributing 
to a complex tone are called its partial tones, or 
simply partials. A partial or partial frequency is 
defined as a definitive energetic frequency band, 
and harmonics or harmonic frequencies are defined as 
partials which are generated in accordance with a 
phenomenon based on an integer relationship such as 
the division of a mechanical object, e.g., a string, 
or of an air column, by an integral number of nodes. 
The tone quality or timbre of a given complex tone 
is determined by the quantity, frequency, and 
amplitude of its disjoint partials, particularly 
their amplitude proportions relative to each other 
and relative frequency to others (i.e., the manner 
in which those elements combine or blend) . 
Frequency alone is not a determining factor, as a 
note played on an instrument has a similar timbre to 
another note played on the same instrument. In 
embodied systems handling sounds, partials actually 
represent energy in a small frequency band and are 
governed by sampling rates and uncertainty issues 
associated with sampling systems. 

Audio signals, especially those relating to 
musical instruments or human voices, have 
characteristic harmonic contents that define how the 
signals sound. Each signal consists of a 
fundamental frequency and higher-ranking harmonic 
frequencies . The graphic pattern for each of these 
combined cycles is the waveform. The detailed 



waveform of a complex wave depends in part on the 
relative amplitudes of its harmonics. Changing the 
amplitude, frequency, or phase relationships among 
harmonics changes the ear's perception of the tone's 
musical quality or character. 

The fundamental frequency (also called the 1st 
harmonic, or fl) and the higher- ranking harmonics 
(f 2 through f„) are often mathematically related. In 
sounds produced by typical musical instruments, 
higher- ranking harmonics are mostly, but not 
exclusively, integer multiples of the fundamental: 
The 2nd harmonic is 2 times the f requency of the 
fundamental, the 3rd harmonic is 3 times the 
frequency of the fundamental, and so on. These 
multiples are often called ranking numbers. In 
general, the usage of the term harmonic in this 
patent represents all harmonics, including the 
fundamental . 

Each harmonic has amplitude, frequency, and 
phase relationships to the fundamental frequency; 
these relationships can be manipulated to alter the 
perceived sound. A periodic complex tone may be 
broken down into its constituent elements 
(fundamental and higher harmonics) . The graphic 
representation of this analysis is called a 
spectrum. A given note's characteristic timbre may 
be represented graphically, then, in a spectral 
profile. 

While typical musical instruments often produce 
notes predominantly containing integer-multiple or 
near integer-multiple harmonics, a variety of other 
instruments and sources produce sounds with more 
complex relationships among fundamentals and higher 



harmonics. Many instruments create harmonics that 
are mathematically related but are purely non- 
integer in their relationship. Furthermore, not all 
instruments produce tones made up of periodic 
vibrations. For tones whose waveforms do not repeat 
each cycle, partial frequencies need not be related 
to the fundamental; these tones are called 
inharraoni ci ti es . 

The modern equal -tempered scale (or Western 
musical scale) is a method by which a musical scale 
is adjusted to consist of 12 equally spaced semitone 
intervals per octave. The frequency of any given 
half-step is the frequency of its predecessor 
multiplied by the 12th root of 2 or 1-0594631. This 
generates a scale where the frequencies of all 
octave intervals are in the ratio 1:2. These octaves 
are the only consonant intervals; all other 
intervals are dissonant. 

The scale's inherent compromises allow a piano, 
for example, to play in all keys. To the human ear, 
however, instruments such as the piano accurately 
tuned to the tempered scale sound quite flat in the 
upper register because harmonics in most mechanical 
instruments are not exact multiples and the "ear 
knows this", so the tuning of some instruments is 
^stretched," meaning the timing contains deviations 
from pitches mandated by simple mathematical 
formulas. These deviations may be either slightly 
sharp or slightly flat to the notes mandated by 
simple mathematical formulas. In stretched tunings, 
mathematical relationships between notes and 
harmonics still exist, but they are more complex. 
The relationships between and among the harmonic 



frequencies generated by. many classes of 
oscillating/ vibrating devices, including musical 
instruments, can be modeled by a function 
f« = f i x G(n) 

where f n is the frequency of the n ch harmonic, 
and n is a positive integer which represents the 
harmonic ranking number. Examples of such functions 
are 

a) f n = f ! x n 

b) f n . f x x n x [1 + (n 2 - 1) 

where Pis constant which depends on the instrument 
or on the string of multiple-stringed devices, and 
sometimes on the frequency register of the note 
being played. 

An audio or musical tone's perceived pitch is 
typically (but not always) the fundamental or lowest 
frequency in the periodic signal. As previously 
mentioned, a musical note contains harmonics at 
various amplitude, frequencies , and phase 
relationships to each other. When super inposed, 
these harmonics create a complex time-domain signal. 
The quantity and amplitude of the harmonics of the 
signal give the strongest indication of its timbre, 
or musical personality. 

Another aspect of an instrument's perceived 
musical tone or character involves resonance bands, 
which are certain fragments or portions of the 
audible spectrum that are emphasized or accented by 
the physics of the instrument. An instrument's 
physics entail aspects such as its design, 
dimensions, materials, construction details, 
features, methods of operation, and response 



characteristics. These resonance bands are perceived 
to be louder relative to other fragments of the 
audible spectrura / and affect the instrument's 
harmonic content. 

Such resonance bands are fixed in frequency and 
remain constant as different notes are played on the 
instrument. These resonance bands do not shift with 
respect to different notes played on the instrument. 
They are determined by the physics of the 
instrument/ not by the particular note played at any 
given time. 

A key difference between harmonic content and 
resonance bands lies in their differing 
relationships to fundamental frequencies. Harmonics 
shift along with changes in the fundamental 
frequency (i.e., they move in frequency, directly 
linked to the played fundamental) and thus are 
always relative to the fundamental. As fundamentals 
shift to new fundamentals, their harmonics shift 
along with them. 

In contrast, an instrument's resonance bands 
are fixed in frequency and do not move linearly as a 
function of shifting fundamentals. 

Aside from a note's own harmonic structure and 
the instrument's own resonance bands, other factors 
contributing to an instrument's perceived tone or 
musical character entail the manner in which 
harmonic content varies over the duration of a 
musical note. The duration or *life span" of a 
musical note is marked by its attack (the 
characteristic manner in which the note is initially 
struck or sounded) ; sustain (the continuing 
characteristics of the note as it is sounded over 



time) ; and decay (the characteristic manner in which 
the note terminates - e.g., an abrupt cut-off vs. a 
gradual fade) , in that order . 

A note's harmonic content during all three 
phases — attack, sustain, and decay — give important 
perceptual keys to the human ear regarding the 
note's subjective tonal quality. Each harmonic in a 
complex time-domain signal, including the 
fundamental, has its own distinct attack and decay 
characteristics, which help define the note's timbre 
in time* 

Because the relative amplitude levels of the 
harmonics may change during the life span of the 
note in relation to the amplitude of the fundamental 
(some being emphasized, some de -emphasized) , the 
timbre of a specific note may accordingly change 
across its duration. In instruments that are 
plucked or struck (such as pianos and guitars) , 
higher-order harmonics decay at a faster rate than 
lower-order harmonics. By contrast, on instruments 
that are continually exercised, including wind 
instruments (such as the flute) and bowed 
instruments (such as the violin) , harmonics are 
continually generated because the source itself 
continually generates signals. 

On a guitar, for example, the two most 
influential factors, which shape the perceived 
timbre, are: (1) the core harmonics created by the 
strings; and (2) the resonance band characteristics 
of the guitar's body. 

Once the strings have generated the fundamental 
frequency and its associated core set of harmonics, 
the body, bridge, and other components come into 
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play to further shape the timbre primarily by its 
resonance characteristics, which are non-linear and 
frequency dependent. A guitar has resonant bands or 
regions, within which some harmonics of a tone are 
emphasized regardless of the frequency of the 
fundamental . 

A guitarist may play the exact same note (same 
frequency, or pitch) in as many as six places on the 
neck using different combinations of string and fret 
positions. However, each of the six versions will 
sound quite distinct due to different relationships 
between the fundamental and its harmonics. These 
differences in turn are caused by variations in 
string composition and design, string diameter 
and/or string length. Here, "length" refers not 
necessarily to total string length but only to the 
vibrating portion which creates musical pitch, i.e., 
the distance from the fretted position to the 
bridge. The resonance characteristics of the body 
itself do not change, and yet because of these 
variations in string diameter and/or length, the 
different versions of the same pitch sound 
noticeably different. 

In many cases it is desired to affect the 
timbre of an instrument. Modern and traditional 
methods do so in a rudimentary form with a kind of 
filter called a fixed-band electronic equalizer. 
Fixed-band electronic equalizers affect one or more 
specified fragments, or bands, within a larger 
frequency spectrum. The desired emphasis ("boost") 
or de-emphasis ("cut") occurs only within the 
specified band. Notes or harmonics falling outside 
the band or bands are not affected. 



A given frequency can have any harmonic ranking 
depending on its relationship relative to the 
changing fundamental. A resonant band filter or 
equalizer recognizes a frequency only as being 
inside or outside its fixed band; it does not 
recognize or respond to that frequency 1 s harmonic 
rank. The device cannot distinguish whether the 
incoming frequency is a fundamental, a 2nd harmonic, 
a 3rd harmonic, etc. Therefore, the effects of 
fixed-band equalizers do not change or shift with 
respect to the frequency's rank. The equalization 
remains fixed, affecting designated frequencies 
irrespective of their harmonic relationships to 
fundamentals. While the equalization affects the 
levels of the harmonics which does significantly 
affect the perceived timbre, it does not change the 
inherent "core" harmonic content of a note, voice, 
instrument, or other audio signal. Once adjusted, 
whether the fixed-band equalizer has any effect at 
all depends solely upon the frequency itself of the 
incoming note or signal. It does not depend upon 
whether that frequency is a fundamental (1st 
harmonic), 2nd harmonic, 3rd harmonic, or some other 
rank. 

Some present day equalizers have the ability to 
alter their filters dynamically, but the alterations 
are tied to time cues rather than harmonic ranking 
information. These equalizers have the ability to 
adjust their filtering in time by changing the 
location of the filters as defined by user input 
commands. One of the methods of the present 
invention, may be viewed as a 1000 -band or more 
graphic equalizer, but is different in that the 
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amplitude and the corresponding affected frequencies 
are instantaneously changing in frequency and 
amplitude and/or moving at very fast speeds with 
respect to frequency and amplitude to change the 
harmonic energy content of the notes; and working in 
unison with a synthesizer adding missing harmonics 
and all following and anticipating the frequencies 
associated with the harmonics set for change. 

The human voice may be thought of as a musical 
instrument, with many of the same qualities and 
characteristics found in other instrument families . 
Because it operates by air under pressure, it is 
fundamentally a wind instrument, but in terms of 
frequency generation the voice resembles a string 
instrument in that multiple-harmonic vibrations are 
produced by pieces of tissue whose vibration 
frequency can be varied by adjusting their tension . 
Unlike an acoustic guitar body, with its fixed 
resonant chamber, some of the voice's resonance 
bands are instantly adjustable because certain 
aspects of the resonant cavity may be altered by the 
speaker, even many times within the duration of a 
single note. Resonance is affected by the 
configuration of the nasal cavity and oral cavity, 
the position of the tongue, and other aspects of 
what in its entirety is called the vocal tract. 

PRIOR ART 

U.S. Patent 5,847,303 to Matsumoto describes a 
voice processing apparatus that modifies the 
frequency spectrum of a human voice input. The 
patent embodies several processing and calculation 
steps to equalize the incoming voice signal so as to 
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make it sound like that of another voice (that of a 
professional singer, for example) . It also provides 
a claim to be able to change the perceived gender of 
the singer. 

The frequency spectrum modification of the 
Mat sumo to Patent is accomplished by using 
traditional resonant band type filtering methods, 
which simulate the shape of the vocal tract or 
resonator by analyzing the original voice* Related 
coefficients for compressor/expander and filters are 
stored in the device's memory or on disk, and are 
fixed (not selectable by the end user) . The 
frequency- following effect of the Matsumoto Patent 
is to use fundamental -frequency information from the 
voice input to offset and tune the voice to the 
"proper" or "correct" pitch. Pitch change is 
accomplished via electronic clock rate manipulations 
that shift the format frequencies within the tract. 
This information is subsequently fed to an 
electronic device which synthesizes complete 
waveforms. Specific harmonics are not synthesized 
not individually adjusted with respect to the 
fundamental frequency, the whole signal is treated 
the same. 

U.S. Patent 5,218,160 to Grob-Da Veiga 
describes a method for enhancing stringed instrument 
sounds by creating undertones or overtones. The 
invention employs a method for extracting the 
fundamental frequency and multiplying that frequency 
by integers or small fractions to create 
harmonically related undertones or overtones. Thus 
the undertones and overtones are derived directly 
from the fundamental frequency. 
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U.S. Patent 5,749,073 to Slaney addresses the 
automatic morphing of audio information. Audio 
morphing is a process of blending two or more 
sounds, each with recognizable characteristics, into 
a new sound with composite characteristics of both 
original sources. 

Slaney uses a multi-step approach. First, the 
two different input sounds are converted to a form 
which allows for analysis, such that they can be 
matched in various ways/ recognizing both harmonic 
relationships and inharmonic relationships. Once 
the inputs are converted, pitch and formant 
frequencies are used for matching the two original 
sounds. Once matched, the sounds are cross- faded 
(i.e., summed, or blended in some pre-selected 
proportion) and then inverted to create a new sound 
which is a combination of the two sounds. The 
method employed uses pitch changing and spectral 
profile manipulation through filtering. As in the 
previously mentioned patents, the methods entail 
resonant type filtering and manipulation of the 
format information. 

Closely related to the Slaney patent is a 
technology described in an article by E. Tellman, L. 
Haken, and B. Holloway titled "Timbre Morphing of 
Sounds with Unequal Numbers of Features" {Journal of 
Audio Engineering Society, Vol. 43, No. 9, Sept. 
1995) . The technology entails an algorithm for 
morphing between sounds using Lemur analysis and 
synthesis. The Tel Iman/Haken/ Holloway timbre- 
morphing concept involves time-scale modifications 
(slowing down or speeding up the passage) as well as 
amplitude and frequency modification of individual 
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sinusoidal (sine wave -based) components. 

U.S. Patent 4, 050,343 by Robert A. Moog relates 
to an electronic music synthesizer. The note 
information is derived from the keyboard key pressed 
by the user. The pressed keyboard key controls a 
voltage/controlled oscillator whose outputs control 
a band pass filter, a low pass filter and an output 
amplifier. Both the center frequency and band width 
of the band pass filters are adjusted by application 
of the control voltage. The low pass cut-off 
frequency of the low pass filter is adjusted by 
application of the control voltage and the gain of 
the amplifier is adjusted by the control voltage. 

In a product called Ionizer [Arboretum 
Systems] f a method starts by using a u pre-analysis" 
to obtain a spectrum of the noise contained in the 
signal - which is only characteristic of the noise. 
This is actually quite useful in audio systems, 
since tape hiss, recording player noise, hum, and 
buzz are recurrent types of noise. By talcing a 
sound print, this can be used as a reference to 
create w anti-noise" and subtract that (not 
necessarily directly) from the source signal. The 
usage of tt peak finding" in the passage within the 
Sound Design portion of the program implements a 
512 -band gated EQ, which can create very steep 
"brick wall" filters to pull out individual 
harmonics or remove certain sonic elements. They 
implement a threshold feature that allows the 
creation of dynamic filters. But, yet again, the 
methods employed do not follow or track the 
fundamental frequency, and harmonic removal again 
must fall in a frequency band, which then does not 
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track the entire passage for an instrument. 

Kyma-5 is a combination of hardware and 
software developed by Symbolic Sound. Kyma-5 is 
software that is accelerated by the Capybara 
hardware platform. Kyma-5 is primarily a synthesis 
tool, but the inputs can be from an existing 
recorded sound files. It has real-time processing 
capabilities, but predominantly is a static-file 
processing tool. An aspect of Kyma-5 is the ability 
to graphically select partials from a spectral 
display of the sound passage and apply processing. 
Kyma-5 approaches selection of the partials visually 
and identifies "connected" dots of the spectral 
display within frequency bands, not by harmonic 
ranking number. Harmonics can be selected if they 
fall within a manually set band. Kyma-5 is able to 
re- synthesize a sound or passage from a static file 
by analyzing its harmonics and applying a variety of 
synthesis algorithms, including additive synthesis. 
However, there is no automatic process for tracking 
harmonics with respect to a fundamental as the notes 
change over time. Kyma-5 allows the user selection 
of one fundamental frequency. Identification of 
points on the Kyma spectral analysis tool may 
identify points that are strictly non-harmonic. 
Finally, Kyma does not apply stretch constants to 
the sounds. 

METHODS AND RESULTS OF INVENTION 
The present invention affects the tonal- 
quality, or timbre, of a signal, waveform, note or 
other signal generated by any source, by modifying 
specific harmonics of each and every fundamental 
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and/ or note, in a user-prescribed manner, as a 
complex audio signal progresses through time. For 
example, the user-determined alterations to the 
harmonics of a musical note (or other signal 
waveform) could also be applied to the next note or 
signal, and to the note or signal after that, and to 
every subsequent note or signal as a passage of 
music progresses through time. It is important to 
note that all aspects of this invention look at 
notes, sounds, partials, harmonics, tones, 
inharraonicities, signals, etc. as moving targets 
over time in both amplitude and frequency and adjust 
the moving targets by moving modifiers adjustable in 
amplitude and frequent over time . 

The invention embodies methods for: 

• dynamically and individually altering the 
energy of any harmonic (f^ through f<») of 
complex waveform; 

• creating new harmonics or inharmonics 
(such as harmonics "missing" from a 
desired sound) with a defined amplitude 
and phase relationship to any other 
harmonics; 

• identifying and imitating naturally 
occurring harmonics in synthesized sounds 
based on integer or user-defined harmonic 
relationships, such as f n m f x x n x 

{s) log 2 (n> ; 

• extracting, modifying, and reinserting 
harmonics into notes; 

• interpolating signals depending on 
frequency, amplitude, and/or other 
parameters to enable adjusting the 
harmonic structure of selected notes, then 
shifting the harmonic structure of signals 
all across the musical range from one of 
those user-adjusted points to the other 
according to any of several user- 
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prescribed curves or contours; 

dynamically altering attack rates, decay 
rates, and/or sustain parameters of 
harmonics ; 

separating any harmonics from a complex 
signal for processing of various types; 

changing the levels of partials within a 
signal based on their frequency and 
amplitude; 

determining harmonic relationships of 
partials and their ranking number from 
higher order harmonic frequencies ; 

continuously changing the levels of a 
complex signal 1 s harmonics based on their 
ranking and amplitude; 

increasing or decreasing harmonics by a 
fixed amount or by variable amounts, 
either throughout an entire selected 
passage, or at any portion within that 
passage; 

restoring characteristic information of 
the source signal that may have been lost, 
damaged, or altered in either the 
recording process or through deterioration 
of original magnetic or other media of 
recorded information; 

calculation of partial and harmonic 
locations using the f n * f x x n x (S) l09 2 {n> 
stretch function; 

harmonically transforming one sound signal 
to match, resemble, or partially resemble 
that of another signal type utilizing 
combinations of the aforementioned 
embodiments of harmonic adjustment and 
harmonic synthesis; 

providing a basis for new musical 
instruments including but not limited to 
new types of guitar synthesizers, bass 
synthesizers, guitars, basses, pianos, 
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keyboards, studio sound-modification 
equipment, mastering sound-modification 
equipment, new styles of equalization 
devices, and new audio digital hardware 
and software technologies pertaining to 
the aforementioned methods to alter a 
note, sound, or signal ; 

Changing the harmonic ranking number parameters 
to amplitude, frequency, and time parameters, 
results may include: 

• separating out or isolating individual 
voices, instruments, musical notes, 
partials, harmonics, sine waves, other 
sounds or signals (or portions of sounds 
or signals) from an aggregation of voices, 
instrumental sounds, or other audio 
signals; 

• highlighting previously hard-to-hear 
voices, instruments, musical notes, 
harmonics, partials, other sounds or 
signals, or portions of sounds or signals, 
within an aggregation of other such 
signals; 

• canceling noise or reducing noise; 

• smoothing out or attenuating previously 
harsh or overly prominent voices, 
instruments, musical notes, harmonics, 
partials, other sounds or signals, or 
portions of sounds or signals, within an 
aggregation of other such signals; 

• enhancing low- volume and/or attenuating or 
diminishing relatively high- volume notes, 
partials, harmonics, inharmonic or other 
signals in a passage of music or other 
complex time-domain signals; 

• eliminate certain amplitude ranges of 
partials such that lower level information 
can be more easily discerned and/or 
processed; 

• and generally effecting a more desirable 
balance of voices, instruments, musical 
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notes, harmonics, partials, other sounds 
or signals, or portions of sounds or 
signals; 

5 

SUMMARY OF METHODS 
This processing is not limited to traditional 
musical instruments, but may be applied to any 
incoming source signal waveform or material to alter 

10 its perceived quality, to enhance particular aspects 

of timbre, or to de-emphasize particular aspects. 
This is accomplished by the manipulation of 
individual harmonics and/or partials of the spectrum 
for a given signal. With the present invention, 

15 adjustment of a harmonics or partials is over a 

finite period of time. This differs from the effect 
of generic, fixed-band equalization, which is 
maintained over an indefinite period of time. 

The assigned processing is accomplished by 

20 manipulating the energy level of a harmonic (or 

group of harmonics) , or by generating a new harmonic 
(or group of harmonics) or partials, or by fully 
removing a harmonic (or group of harmonics) or 
partials. The manipulations can be tied to the 

25 response of any other harmonic or it can be tied to 

any frequency or ranking number (s) or other 
parameter the user selects. This is eminently 
practical because, to cite but one example of 
harmonic behavior, the adjusted harmonics will die 

30 out relatively slowly if the frequency to which they 

are tied dies out relatively slowly. Conversely, 
adjusted harmonics will die out relatively quickly 
if the frequency to which they are tied dies out 
relatively quickly. Adjustments can also be 

35 generated independently of existing harmonics. In 
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some cases, multiple manipulations using any 
combination of methods may be used. In others, a 
harmonic or group of harmonics may be separated out 
for individual processing by various means. In 
still others, partials can be emphasized or de- 
emphasized. 

The preferred embodiment of the manipulation of 
the harmonics uses Digital Signal Processing (DPS) 
techniques. Filtering and analysis methods are 
carried out on digital data representations by a 
computer (e.g. DSP or other microprocessor) . The 
digital data represents an analog signal or complex 
waveform that has been sampled and converted from an 
analog electrical waveform to digital data. Upon 
con£>letion of processing, the data may be converted 
back to an analog electrical signal. It also may be 
transmitted in a digital form to another system, as 
well as being stored locally on some form of 
magnetic or other storage media. The signal sources 
are quasi real-time or prerecorded in a digital 
audio format, and software is used to carry out the 
desired calculations and manipulations. 

Other objects, advantages and novel features of 
the present invention will become apparent from the 
following detailed description of the invention when 
considered in conjunction with the accompanying 
drawings . 

DESCRIPTION OF DRAWINGS 
Figure 1 is four graphs of four notes and four 
of its harmonics on a frequency versus amplitude 
scale showing the accordion effect of harmonics as 
they relate to each other. 
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Figure 2 is a graph of the harmonic content of 
a note at a particular point in time on a frequency 
versus amplitude scale. 

Figure 3 is an adjustment of the individual 
frequencies and synthesized frequencies of the note 
of Figure 2 incorporating the principles of the 
present invention. 

Figure 4 is a schematic of a first embodiment 
of a system for performing the method illustrated in 
Figure 3 using an amplitude and frequency following 
filter method according to the present invention. 

Figure 5 is a block diagram of a system for 
performing the method of Figure 3 using a bucket 
brigade method according to the present invention. 

Figure 6 is a spectral profile graph of a 
complex waveform from a single strike of a 440 Hertz 
piano keys as a function of frequency (X axis) , time 
(Y axis) , and magnitude (Z axis) . 

Figure 7 is a graph of a signal modified 
according to the principles of Harmonic and other 
Partial accentuation and/or Harmonic 
Transformations . 

Figures 8A, 8B, 8C and 8D illustrate the 
spectral content of a flute and piano at times both 
early and late in the same note as it relates to 
Harmonic Transformation. 

Figure 9A is a graph showing potential 
threshold curves for performing an accentuation 
method according to the present invention. 

Figure 9B is a graph illustrating potential 
low levels of adjustment to be used with Figure 9A. 

Figure 9C is a graph illustrating a potential 
fixed- threshold method of Harmonic and other Partial 



Accentuation . 

Figure 9D is a graph illustrating a frequency 
band dynamic threshold example curve for one method 
of Harmonic and other Partial Accentuation. 

Figure 10 is a block diagram of a system for 
performing the operations of the present invention. 

Figure 11 is a block diagram of the software or 
method steps incorporating the principles of the 
present invention. 

DETAILED DESCRIPTI ON OF THE PREFERRED EMBODIMENTS 
HARMONIC ADJUSTMENT 

The goal of harmonic adjustment and synthesis 
is to manipulate the characteristics of harmonics on 
an individual basis based on their ranking numbers. 
The manipulation is over the time period that a 
particular note has amplitude. A harmonic may be 
adjusted by applying filters centered at its 
frequency. Throughout this invention, a filter may 
also be in the form of an equalizer, mathematical 
model, or algorithm. The filters are calculated 
based on the harmonic's location in frequency, 
amplitude, and time with respect either to any other 
harmonic. Again, this invention looks at harmonics 
as moving frequency and amplitude targets. 

One aspect of the present invention is its 
ability to anticipate changes in audio signals 
including (but not limited to) changes in harmonic 
frequency, amplitudes, phase, harmonic content, and 
other aspects described herein, and to make 
adjustments accordingly in response to such changes. 
This is irrespective of how the signal was stored, 
such as whether the original audio signal was stored 
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on magnetic tape, a digital device, or any other 
media. It is also irrespective of the time, such as 
a few milliseconds, seconds, minutes, days, weeks, 
months or years prior to the present invention's 
processing. In most cases, the present invention 
"looks ahead" to all manners of shifts in upcoming 
signals and reacts according to calculation and user 
input and control. 

"Looking ahead* in quasi real-time actually 
entails collecting data for a minimum amount of 
time such that appropriate characteristics of the 
incoming data (i.e. audio signal) may be recognized 
to trigger appropriate processing. This information 
is stored in a delay buffer until needed aspects are 
ascertained. The delay buffer is continually being 
filled with new data and unneeded data is removed 
from the "oldest" end of the buffer when it is no 
longer needed. This is how a small latency occurs 
in quasi real-time situations. 

Quasi -real time refers to a minuscule delay of 
up to approximately 60 milliseconds. It is often 
described as about the duration of up to two frames 
in a motion-picture film, although one frame delay 
is preferred. 

In the present invention the processing filters 
anticipate the movement of and move with the 
harmonics as the harmonics move with respect to 
frequency and amplitude. Thus, as the first 
harmonic (fi) changes, the designated harmonic (or 
"harmonic set for amplitude adjustment") will shift 
in frequency by mathematically fixed amounts related 
to the harmonic ranking. For example, if the first 
harmonic (f x ) changes from 100 Hz to 110 Hz, the 
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present invention's harmonic adjustment filter for 
the fourth harmonic <f 4 ) shifts from 400 Hz to 440 
Hz. 

Figure 1 shows a series of four notes and the 
characteristic harmonic content of four harmonics of 
each note at a given point in time. This 
hypothetical sequence shows how the harmonics and 
filters move with respect to the fundamental, the 
harmonics, and with respect to each other. The 
tracking of these moving harmonics in both amplitude 
and frequency over time is a key element in the 
processing methods embodied herein. 

The separation or distance between frequencies 
{corresponding to the separation between filters) 
expands as fundamentals rise in frequency, and 
contracts as fundamentals lower in frequency. 
Graphically speaking, this process is to be known 
herein as the "accordion effect." 

The present invention is designed to adjust 
amplitudes of harmonics over time with filters which 
move with the non-stationary (frequency changing) 
harmonics of the signals set for amplitude 
adjustment. 

Specifically, the individual harmonics are 
parametrically filtered and/or amplified. This 
increases and decreases the relative amplitudes of 
the various harmonics in the spectrum of individual 
played notes based not upon the frequency band in 
which the harmonics appear (as is presently done 
with conventional devices) , but rather based on 
their harmonic ranking numbers and upon which 
harmonic ranks are set to be filtered. This may be 
done off-line, for example, after the recording of 
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music or complex waveform, or in quasi -real time. 
For this to be done in quasi-real time, the 
individual played note's harmonic frequencies are 
determined from the instrument's spectral signature 
using a common frequency detection method or Fast 
Find Fundamental (to be described below) , and the 
harmonic -by-harmonic filtering is then performed on 
the determined notes. 

Because harmonics are being manipulated in this 
unique fashion, the overall timbre of the instrument 
is affected with respect to individual, precisely 
selected harmonics , as opposed to merely affecting 
fragments of the spectrum with conventional filters 
assigned to one or more fixed resonance bands. 

For the ease of illustration, the model of the 
harmonic relationship in Figures 1-3 will be 
f n « f x x n. 

For example, this form of filtering will filter 
the 4th harmonic at 400Hz the same way that it 
filters the 4th harmonic at 2400Hz, even though the 
4th harmonics of those two notes (note 1 and note 3 
of Figure 1) are in different frequency ranges. 
This application of the present invention will be 
useful as a complement to, and/or a replacement for, 
conventional frequency-band-by- frequency-band 
equalization devices. The mixing of these 
individually filtered harmonics of the played notes 
for output will be discussed with respect to Figures 
4 and 5. 

Figure 2 shows an example of the harmonic 
content of a signal at a point in time. The 
fundamental frequency (f x ) is 100 Hz. Thus, in 
multiples of 100 Hz, one sees the harmonics of this 
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signal at 200 Hz (f^f^)., 300 Hz (f 2 ^f{x:3) , 400 Hz 
(f 4 =fiX4) , etc. For illustration, this example has a 
total of 10 harmonics , but actual signals often have 
many more harmonics* 

Figure 3 shows the adjustment modif ication, as 
could be effected with the present invention, of 
some harmonics of Figure 2. Harmonics located at 200 
Hz (2nd harmonic) , 400 Hz (4th harmonic) , 500 Hz 
(5th) , and 1000 Hz (10th) are all adjusted upwards 
in energy content and amplitude. Harmonics at 600 
Hz (6th harmonic), 700 Hz (7th harmonic), 800 Hz 
(8th) , and 900 Hz (9th) are all adjusted downward in 
energy content and amplitude. 

With the present invention, harmonics may be 
either increased or decreased in amplitude by 
various methods. One present-day method is to apply 
specifically calculated digital filters over the 
time frame of interest. These filters adjust their 
amplitude and frequency response to move with the 
harmonic's frequency being adjusted. Other methods 
also employ Digital Signal Processing, such as 
matching the phase of sinusoids to a harmonic of 
interest, then (A) subtracting the desired amount by 
adding an inverse of that waveform to the original 
signal, for reduction; or (B) adding a scaled 
version (that is, one which has been multiplied by 
some designated factor) , for enhancement . 

Other embodiments may utilize a series of 
filters adjacent in frequency or a series of fixed 
frequency filters, where the processing is handed 
off in a "bucket -brigade" fashion as a harmonic 
moves from one filter's range into the next filter's 
range . 
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Figure 4 shows an implementation embodiment. 
The signal at input 10 , which may be from a pickup, 
microphone or pre -stored data, is provided to a 
harmonic signal detector HSD 12 and to a bank of 
filters 14 . Each of the filters in the bank 14 is 
programmable for a specific harmonic frequency of 
the harmonic detected signal and is represented by 
■£ 1# f 2 , f 3 ... f M . A controller 16 adjusts the 
frequency of each of the filters to the frequency 
which matches the harmonic frequency detected by 
harmonic signal detector 12 for its ranking. The 
desired modification of the individual harmonics is 
controlled by the controller 16 based on user 
inputs. The output of the bank of filters 14 are 
combined in mixer 18 with the input signal from 
input 10 and provided as combined output signal at 
output 20 dependent upon the specific algorithm 
employed. As will be discussed with respect to 
Figure 3 below, the controller 16 may also provide 
synthetic harmonics at the mixer 18 to be combined 
with the signal from the equalizer bank 14 and the 
input 10. 

Figure 5 shows the system modified to perform 
the alternate bucket brigade method. The equalizer 
bank 14' has a bank of filters, each having a fix 
frequency adjacent band width represented by Fa, Fb, 
Fc, etc. The controller 16, upon receipt of the 
harmonic signal identified by. the harmonic signal 
detector 12 adjusts the signal modification of the 
characteristic of the fixed band width filters of 
14' to match that of the detected harmonic signals. 
Wherein the filters in bank 14 of Figure 4 each has 
its frequency adjusted to and its modification 



characteristics fixed for the desired harmonic, the 
equalizers of bank 14' of Figure 5 each have their 
frequency fixed and their modification 
characteristics varied depending upon the detected 
harmonic signal* 

Whether employing the accordion frequency and 
amplitude adjustable moving filter method or bucket - 
brigade method of frequency anticipated frequency 
following, or a combination of these methods, the 
filtering effect moves in frequency with the 
harmonic set for amplitude change, responding not 
merely to a signal's frequency but to its harmonic 
rank and amplitude. 

Although the harmonic signal detector 12 is 
shown separate from the controller 16, both may be 
software in a common DSP or microcomputer. 

Preferably, the filters 14 are digital. One 
advantage of digital filtering is that undesired 
shifts in phase between the original and processed 
signals, called phase distortions, can be minimized. 
In one method of the present invention, either of 
two digital filtering methods may be used, depending 
on the desired goal: the Finite Impulse Response 
(FIR) method, or the Infinite Impulse Response (IIR) 
method. The Finite Impulse Response method employs 
separate filters for amplitude adjustment and for 
phase compensation. The amplitude adjustment 
filter (s) may be designed so that the desired 
response is a function of an incoming signal's 
frequency. Digital filters designed to exhibit such 
amplitude response characteristics inherently affect 
or distort the phase characteristics of a data 
array . 
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As a result, the amplitude adjustment filter is 
followed by a second filter placed in series, the 
phase compensation filter. Phase compensation 
filters are unity-gain devices, that counteract 
5 phase distortions introduced by the amplitude 

adjustment filter. 

Filters and other sound processors may be 
applied to either of two types of incoming audio 
signals: real-time, or non-real-time (fixed, or 

10 static) . Real-time signals include live 

performances , whether occurring in a private 
setting, public arena, or recording studio. Once the 
complex waveform has been captured on magnetic tape, 
in digital form, or in some other media, it is 

15 considered fixed or static; it may be further 

processed. 

Before digital processing can be applied to an 
incoming signal, that input signal itself must be 
converted to digital inf ormation. An array is a 
20 sequence of numbers indicating a signal f s digital 

representation. A filter may be applied to an array 
in a forward direction, from the beginning of the 
array to the end; or backward, from the end to the 
beginning . 

25 In a second digital filtering method, Infinite 

Impulse Response (IIR) , zero-phase filtering may be 
accomplished with non-real-time (fixed, static) 
signals by applying filters in both directions 
across the data array of interest. Because the phase 

30 distortion is equal in both directions, the net 

effect is that such distortion is canceled out when 
the filters are run in both directions. This method 
is limited to static (fixed, recorded) data. 



One method of this invention utilizes high-speed 
digital computation devices as well as methods of 
quantifying digitized music, and improves 
mathematical algorithms for adjuncts for high-speed 
Fourier and/or Wavelet Analysis. A digital device 
will analyze the existing music, adjust the 
harmonics 1 volumes or amplitudes to desired levels. 
This method is accomplished with very rapidly 
changing, complex pinpoint digital equalization 
windows which are moving in frequency with harmonics 
and the desired harmonic level changes as described 
in Figure 4. 

The applications for this invention can be 
applied to and not limited to guitars, basses, 
pianos, equalization and filtering devices, 
mastering devices used in recording, electronic 
keyboards, organs, instrument tone modifiers, and 
other waveform modifiers. 

In many situations where it is desired to 
adjust the energy levels of a musical note's or 
other audio signal «s harmonic content, it may 
impossible to do so if the harmonic content is 
intermittent or effectively nonexistent. This may 
occur when the harmonic has faded out below the 
noise "floor" (minimum discernible energy level) of 
the source signal. With the present invention, these 
missing or below- floor harmonics may be generated 
"from scratch," i.e., electronically synthesized. 
It might also be desirable to create an entirely new 
harmonic, inharmonic, or sub-harmonic (a harmonic 
frequency below the fundamental) altogether, with 
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multiplier relationship to the source signal. Again, 
this creation or generation process is a type of 
synthesis. Like naturally occurring harmonics, 
5 synthesized harmonics typically relate 

mathematically to their fundamental frequencies. 

As in Harmonic Adjustment, the synthesized 
harmonics generated by the present invention are 
non- stationary in frequency: They move in relation 

10 to the other harmonics . They may be synthesized 

relative to any individual harmonic (including f^ 
and moves in frequency as the note changes in 
frequency, anticipating the change to correctly 
adjust the harmonic synthesizer. 

15 As shown in Figure 2, the harmonic content of 

the original signal includes frequencies up to 1000 
Hz (10th harmonic of the 100 Hz fundamental) ; there 
are no 11th or 12th harmonics present* Figure 3 
shows the existence of these missing harmonics as 

20 created via Harmonic Synthesis. Thus, the new 

harmonic spectrum includes harmonics up to 12 00 Hz 
(12th harmonic) . 

Instruments are defined not only by the 
relative levels of the harmonics in their audible 

25 spectra but also by the phase of the harmonics 

relative to fundamentals (a relationship which may 
vary over time) . Thus, Harmonic Synthesis also 
allows creation of harmonics which are both 
amplitude -correlated and phase-aligned (i.e., 

3 0 consistently rather than arbitrarily matched to, or 

related to, the fundamental) . Preferably, the bank 
of filters 14 and 14' are digital devices which are 
also digital sine wave generators, and preferably, 



the synthetic harmonics are created using a function 
other than fn ■ f t x a, The preferred relationship 
is for generating the new harmonics f n = f a x n x 
(S) lcg 2 (n> . S is a number greater than 1, for example, 
1.002. This relationship has been found to be very 
accurate for instruments, including the human voice. 

HARMONIC ADJUS TMENT AND SYNTHESIS 
Combinations of Harmonic Adjustment and 
Synthesis embody the ability to dynamically control 
the amplitude of all of the harmonics contained in a 
note based on their ranking, including those 
considered to be tt missing" . This ability to control 
the harmonics gives great flexibility to the user in 
manipulating the timbre of various notes or signals 
to his or her liking. The method recognizes that 
different manipulations may be desired based on the 
level of the harmonics of a particular incoming 
signal. It embodies Harmonic Adjustment and 
Synthesis. The overall timbre of the instrument is 
affected as opposed to merely affecting fragments of 
the spectrum already in existence. 

It may be impossible to adjust the energy 
levels of a signal" s harmonic content if that 
content is intermittent or effectively nonexistent, 
as when the harmonic fades out below the noise 
"floor" of the source signal. With the present 
invention, these missing or below- floor harmonics 
may be generated "from scratch," or electronically 
synthesized, and then mixed back in with the 
original and/or harmonically adjusted signal. 

To address this, Harmonic Synthesis may also be 
used in conjunction with Harmonic Adjustment to 
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alter the overall harmonic, response of the source 
signal. For example, the 10th harmonic of an 
electric guitar fades away much faster than lower 
ranking harmonics, as illustrated in Figure 6. It 
5 might be of interest to use synthesis not only to 

boost the level of this harmonic at the initial 
portion of the note but also to maintain it 
throughout the note's entire existence. The 
synthesis may be carried on throughout all of the 

10 notes in the selected sections or passages. Thus, an 

existing harmonic may be adjusted during the portion 
where it exceeds a certain threshold, and then 
synthesized (in its adjusted form) during the 
remaining portion of the note (see Figure 7) . 

15 It may also be desired to accomplish this for 

several harmonics. In this case, the harmonic is 
synthesized with desired phase- alignment to maintain 
an amplitude at the desired threshold. The phase 
alignment may be drawn from an arbitrary setting, or 

20 the phase may align in some way with a user-selected 

harmonic. This method changes in frequency and 
amplitude and/or moves at very fast speeds to change 
the harmonic energy content of the notes and works 
in unison with a synthesizer to add missing desired 

25 harmonics. These harmonics and synthesized harmonics 

will be proportional in volume to a set harmonic 

amplitude at percentages set in a digital device's 

software. Preferably, the function f n » f 1 x n x 

(S) l03 2 (n) is used to generate a new harmonic. 
3 0 In order to avoid the attempted boosting of a 

harmonic that does not exist, the present invention 

employs a signal -detect ion algorithm to indicate 

that there is enough of a signal present to make 
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warranted adjustments. Typically, such signal 
detection methods are based on the total energy of 
the input signal, such that as long as the signal 
energy (or amplitude) is above a threshold for some 
arbitrarily defined time period, it is considered to 
be present. 

HARMONIC TRANSFORM&T ION 
Harmonic Transformation refers to the present 
invention' 9 s ability to compare one sound or signal 
(the file set for change) to another sound or signal 
(the second file), and then to employ Harmonic 
Adjustment and Harmonic Synthesis to adjust the 
signal set for change so that it more closely 
resembles the second file or, if desired, duplicates 
the second file in timbre. These methods combines 
several aspects of previously mentioned inventions 
to accomplish an overall goal of combining audio 
sounds, or of changing one sound to more closely 
resemble another. It can be used, in fact, to make 
one recorded instrument or voice sound almost 
exactly like another instrument or voice. 

When one views a given note produced by an 
instrument or voice in terms of its harmonic 
frequency content with respect to time (Figure 6) , 
one sees that each harmonic has an attack 
characteristic (how fast the initial portion of that 
harmonic rises in time and how it peaks), a sustain 
characteristic (how the harmonic structure behaves 
after the attack portion) , and a decay 
characteristic (how the harmonic stops or fades away 
at the end of a note) . In some cases, a particular 
harmonic may have faded completely away before the 
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fundamental itself has ended. 

Different examples of one type of musical 
instrument (two pianos, for example) can vary in 
many ways. -One variation is in the harmonic content 
of a particular complex time-domain signal. For 
example, a middle "C" note sounded on one piano may 
have a very different harmonic content than the same 
note sounded on a different piano. 

Another way in which two pianos can differ 
refers to harmonic content over time. Not only will 
the same note played on two different pianos have 
different harmonic structures, but also those 
structures will behave in different ways over time. 
Certain harmonics of one note will sustain or fade 
out in very different manners compared to the 
behavior over time of the harmonic structure of the 
same note sounded on a different piano. 

By individually manipulating the harmonics of 
each signal produced by a recorded instrument, that 
instrument's response can be made to closely 
resemble or match that of a different instrument. 
This technique is termed harmonic transformation. It 
can consist of dynamically altering the harmonic 
energy levels within each note and shaping their 
energy response in time to closely match harmonic 
energy levels of another instrument. This is 
accomplished by frequency band comparisons as it 
relates to harmonic ranking. Harmonics of the first 
file (the file to be harmonically transformed) are 
compared to a target sound file to match the attack, 
sustain, and decay characteristics of the second 
file's harmonics. 

Since there will not be a one-to-one match of 
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harmonics, comparative analysis will be required by 
the algorithm to create rules for adjustments- This 
process can also be aided by input from the user 
when general processing occurs* 

An example of such manipulation can be seen 
with a flute and piano. Figures 8a through 8d show 
spectral content plots for the piano and the flute 
at specific points in time. Figure 8a shows the 
spectral content of a typical flute early in a note. 
Figure 8b shows the flute 1 s harmonic content much 
later in the same note. Figure 8c shows the same 
note at the same relative point in time as 8a from a 
typical piano. At these points in time, there are 
large amounts of upper harmonic energy. However, 
later in time, the relative harmonic content of each 
note has changed significantly. Figure 8d is at the 
same relative point in time for the same note as 8b, 
but on the piano. The piano 1 s upper harmonic content 
is much sparser than that of the flute at this point 
in the note. 

Since one sound file can be made to more 
closely resemble a vast array of other sound 
sources, the information need not come directly from 
a second sound file. A model may be developed via a 
variety of means. One method would be to general 
characterize another sound based on its behavior in 
time, focusing on the characteristic harmonic or 
partial content behavior. Thus, various 
mathematical or other logical rules can be created 
to guide the processing of each harmonic of the 
sound file that is to be changed. The model files 
may be created from another Sound file, may be 
completely theoretical models, or may, in fact, be 



36 

arbitrarily defined by a user. 

Suppose a user wishes to make a piano sound 
like a flute; this process requires considering the 
relative characteristics of both instruments- A 
piano has a large burst of energy in its harmonics 
at the outset of a note, followed by a sharp fall- 
off in energy content. In comparison, a flute 1 s 
initial attack is less pronounced and has 
inharmonicities. With the present invention, each 
harmonic of the piano would be adjusted accordingly 
during this phase of every note so as to approximate 
or, if needed, synthesize corresponding harmonics 
and missing partials of the flute ♦ 

During the sustain portion of a note on a 
piano, its upper harmonic energy content dies out 
quickly, while on a flute the upper harmonic energy 
content exists throughout the duration of the note. 
Thus, during this portion, continued dynamic 
adjustment of the piano's harmonics is required. In 
fact, at some point, synthesis is required to 
replace harmonic content when the harmonics drop to 
a considerably lower level. Finally, on these two 
instruments the decay of a note is slightly 
different as well, and appropriate adjustment is 
again needed to match the flute. 

This is achieved by the usage of digital 
filters, adjustment parameters, thresholds, and sine 
wave synthesizers which are used in combination and 
which move with or anticipate shifts in a variety of 
aspects of signals or notes of interest, including 
the fundamental frequency. 
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In the present invention, Harmonic and other 
Partial Accentuation provides a method of adjusting 
sine waves, partials, inharmonicities, harmonics, or 
other signals based upon their amplitude in relation 
to the amplitude of other signals within associated 
frequency ranges. It is an alteration of harmonic 
adjustment using amplitudes in a frequency range to 
replace harmonic ranking as a filter amplitude 
position guide or criteria. Also, as in Harmonic 
Adjustment, the partial 's frequencies are the 
filters frequency adjusting guide because partials 
move in frequency as well as amplitude. Among the 
many audio elements typical of musical passages or 
other complex audio signals, those which are weak 
may, with the present invention, be boosted relative 
to the others, and those which are strong may be cut 
relative to the others, with or without compressing 
their dynamic range as selected by the user. 

The present inventions (1) isolate or highlight 
relatively quiet sounds or signals; (2) diminish 
relatively loud or other selected sounds or signals, 
including among other things background noise, 
distortion, or distracting, competing, or other 
audio signals deemed undesirable by the user; and 
(3) effect a more intelligible or otherwise more 
desirable blend of partials, voices, musical notes, 
harmonics, sine waves, other sounds or signals; or 
portions of sounds or signals . 

Conventional electronic compressors and 
expanders operate according to only a very few of 
the parameters which are considered by the present 
invention, and by no means all of them. 
Furthermore, the operation of such 
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compression/expansion devices is fundamentally 
different than that of the present invention. With 
Accentuation, the adjustment of a signal is based 
not only upon its amplitude but can also be by its 
5 amplitude relative to amplitudes of other signals 

within its frequency range. For example, the sound 
of feet shuffling across a floor may or may not need 
to be adjusted in order to be heard. In an otherwise 
quiet room the sound may need no adjustment, whereas 

10 the same sound at the same amplitude occurring 

against a backdrop of strongly competing partials, 
sounds or signals may require accentuation in order 
to be heard. The present invention can make such a 
determination and act accordingly. 

15 In one method of the present invention, a piece 

of music is digitized and amplitude modified to 
accentuate the quiet partials. Present technology 
accomplishes this by compressing the music in a 
fixed frequency range so that the entire signal is 

20 affected based on its overall dynamic range. The 

net effect is to emphasize quieter sections by 
amplifying the quieter passages. This aspect of the 
present invention works on a different principle. 
Computer software examines a spectral range of a 

25 complex waveform and raises the level of individual 

partials that are below a particular set threshold 
level. Likewise, the level of partials that are 
above a particular threshold may be lowered in 
amplitude. Software will examine all partial 

30 frequencies in the complex waveform over time and 

modify only those within the thresholds set for 
change. In this method, analog and digital hardware 
and software will digitize music and store it in 
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some form of memory. The complex waveforms will be 
examined to a high degree of accuracy with Past 
Fourier Transforms, wavelets , and/or other 
appropriate analysis methods. Associated software 
will compare over time calculated partials to 
amplitude/ frequency, and time thresholds and/or 
parameters, and decide which partial frequencies 
will be within the thresholds for att^litude 
modification. These thresholds are dynamic and are 
dependent upon the competing partials surrounding 
the partial slated for adjustment within some 
specified frequency range on either side. 

This part of the present invention acts as a 
sophisticated, frequency- selective equalization or 
filtering device where the number of frequencies 
that can be selected will be almost unlimited. 
Digital equalization windows will be generated and 
erased so that partials in the sound that were hard 
to hear are now more apparent to the listener by 
modifying their start, peak, and end amplitudes. 

As the signal of interest's amplitude shifts 
relative to other signals' amplitudes, the 
flexibility of the present invention allows 
adjustments to be made either (1) on a continuously 
variable basis, or (2) on a fixed, non- continuously 
variable basis. The practical effect is the ability 
not only to pinpoint portions of audio signals that 
need adjustment and to make such adjustments, but 
also to make them when they are needed, and only 
when they are needed. Note that if the filter 
changes are faster than about 30 cycles per second, 
they will create their own sounds. Thus, changes at 
a rate faster than this are not proposed unless low 



bass sounds can be filtered out. 

The present invention's primary method (or 
combinations thereof) entails filters that move in 
frequency and amplitude according to what's needed 
to effect desired adjustments to a particular 
partial (or a fragment thereof) at a particular 
point in time. 

In a secondary method of the present invent ion, 
the processing is "handed off" in a « bucket -brigade" 
fashion as the partial set for amplitude adjustment 
moves from one filters range into the next filter 1 s 
range . 

As stated previously, both methods of the 
present invention entail the ability to "look ahead" 
i.e. anticipate in non-real time or in quasi -real 
time, both the amplitude as well as the frequency or 
harmonic content of partials to be adjusted (or 
portions thereof) which comprise a complex audio 
signal. In doing so, the present invention can 
examine frequency, frequency over time, competing 
partials in frequency bands over time, amplitude, 
and amplitude over time. Then, with the use of 
frequency and amplitude adjustable filters, 
mathematical models, or algorithms, it dynamically 
adjusts the amplitudes of those partials, harmonics, 
or other signals (or portions thereof) as necessary 
to achieve the goals, results or effects as 
described above. In both methods, after assessing 
the frequency and amplitude of a partial, other 
signals, or portion thereof, the present invention 
determines whether to adjust the signal up, down, or 
not at all, based upon thresholds. 

Accentuation relies upon amplitude thresholds 
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and adjustment curves. There are three methods of 
implementing thresholds and adjustments in the 
present invention to achieve desired results. The 
first method utilizes a threshold that dynamically 
adjusts the amplitude threshold based on the overall 
energy of the complex waveform. The energy 
threshold maintains a consistent frequency 
dependence (i.e. the slope of the threshold curve is 
consistent as the overall energy changes) . The 
second method implements an interpolated threshold 
curve within a frequency band surrounding the 
partial to be adjusted. The threshold is dynamic 
and is localized to the frequency region around this 
partial . The adjustment is also dynamic in the same 
f requency band and changes as the surrounding 
partials within the region change in amplitude. 
Since a partial may move in frequency, the threshold 
and adjustment frequency band are also frequency- 
dynamic, moving with the partial to be adjusted as 
it moves. The third utilizes a fixed threshold 
level. Partials whose amplitude are above the 
threshold are adjusted downward. Those below the 
threshold and above the noise floor are adjusted 
upwards in amplitude. These three methods are 
discussed below. 

In all three methods, the adjustment levels are 
dependent on a "scaling function". When a harmonic 
or partial exceeds or drops below a threshold, the 
amount it exceeds or drops below the threshold 
determines the extent of the adjustment. For 
example, a partial that barely exceeds the upper 
threshold will only be adjusted downward by a small 
amount, but exceeding the threshold further will 
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cause a larger adjustment to occur* The transition 
of the adjustment amount is a continuous function. 
The simplest function would be a linear function, 
but any scaling function may be applied. As with 
5 any mathematical function, the range of the 

adjustment of the partials exceeding or dropping 
below the thresholds may be either scaled or offset. 
When the scaling function effect is scaled, the same 
amount of adjustment occurs when a partial exceeds a 

10 threshold, regardless of whether the threshold has 

changed. For example, in the first method listed 
above, the threshold changes when there is more 
energy in the waveform. The scaling function may 
still range between 0% and 25% adjustment of the 

15 partial to be adjusted, but over a smaller amplitude 

range when there is more energy in a waveform. An 
alternative to this is to just offset the scaling 
function by some percentage. Thus, if more energy 
is in the signal, the range would not be the same. 

20 it may now range from 0% to only 10%, for example. 

But, the amount of change in the adjustment would 
stay consistent relative to the amount of energy the 
partial exceeded the threshold. 

By following the first threshold and adjustment 

25 method, it may be desirable to affect a portion of 
the partial content of a signal by defining minimum 
and maximum limits of amplitude. Ideally, such 
processing keeps a signal within the boundaries of 
two thresholds: an upper limit, or ceiling; and a 

30 lower limit, or floor. Partial 's amplitudes are not 

permitted to exceed the upper threshold or to fall 
beneath the lower threshold longer than a set 
period. These thresholds are frequency- dependent as 
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illustrated in Figure 9A. A noise floor must be 
established to prevent the adjustment of partials 
that are actually just low-level noises. The noise 
floor acts as an overall lower limit for 
accentuation consideration and may be established 
manually or through an analysis procedure. 
Coexistent with the frequency- dependent threshold 
curves is frequency- dependent adjustment curves. 
Each incoming partial may be compared to the two 
threshold curves , then adjusted upwards (boosted in 
energy) , downwards (decreased in energy) , or not at 
all. Because any necessary boosts or cuts are 
relative to the overall signal amplitude in the 
partial 's frequency range, the threshold curves 
likewise vary depending upon the overall signal 
energy at any given point in time. If a fixed 
adjustment of a partial 's amplitude were maintained 
above or below a threshold, it would be likely that 
the adjusted partial may become lower than other 
partials surrounding the partial to be adjusted. To 
avoid this, adjustment amounts vary according to the 
level of the partial at a particular point in its 
existence. As discussed above, the adjustment occurs 
based on the scaling function. The adjustment then 
varies dependent upon the amount of energy that the 
partial to be adjusted exceeds or drops below the 
threshold. 

In the second threshold and adjustment method, 
a partial is compared to Competing" partials in a 
frequency band surrounding the partial to be 
adjusted in the time period of the partial. This 
frequency band has several features. These are shown 
in Figure 9D. 1) The width of the band can be 
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modified according to the desired results. For 
example, the band may represent 75% of the frequency 
below the partial and 125% above the partial. 2) 
The shape of the threshold and adjustment region is 
a continuous curve, and is graduated to meet the 
"linear" portion of the overall curve. The linear 
portion of the curve represents the frequencies 
outside of the comparison and adjustment region for 
this partial. However, the overall "offset" of the 
linear portion of the curve is dependent upon the 
overall energy in the waveform. Thus, one may see 
an overall shift in the offset of threshold, but the 
adjustment of the particular partial may not change, 
since it's adjustment is dependent upon the partials 
in its own frequency region. The upper threshold in 
the frequency band of comparison raises with 
competing partials. The scaling function for the 
adjustment of a partial above the threshold line 
shifts or re-scales as well. The lower threshold in 
the frequency band of comparison lowers with 
competing partials. Again, the scaling function for 
the adjustment of a partial shifts or re-scales as 
well. 3) Adjustment amounts are graduated. When a 
partial exceeds or drops below the threshold, its 
adjustment is dependent upon how much the amplitude 
exceeds or drops below the threshold. The 
adjustment amount is a continuous parameter that is 
also offset by the energy in the competing partials 
surrounding the partial being followed. For 
example, if the partial barely exceeds the upper 
threshold, it may be adjusted downward in amplitude 
by only, say, 5%. A more extreme case may see that 
partial adjusted by 25% if its amplitude were to 
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exceed the upper threshold by a larger amount. 
However, if the overall signal energy were 
different, this adjustment amount would be offset by 
some percentage, relating to an overall shift in the 
5 threshold offset. 4) A noise floor must be 

established to prevent the adjustment of partials 
that are actually just low-level noises. The noise 
floor acts as an overall lower limit for 
accentuation consideration and may be established 
10 manually or through an analysis procedure. 

In the third threshold and adjustment method, 
all of the same adjustment methods are employed, but 
the comparison is made to a single fixed threshold. 
In the previous methods, the thresholds were 
15 dependent either upon the overall energy of the 

waveform, or were dependent upon harmonics or 
partials within a frequency band surrounding the 
harmonic or partial to be adjusted. Figure 9c shows 
an example of such a threshold. This method also 
20 uses adjustment amounts that are graduated. When a 
partial exceeds or drops below the threshold, its 
adjustment is dependent upon how much the amplitude 
exceeds or drops below the threshold. The 
adjustment amount is a continuous parameter that is 
25 also offset or re-scaled by the energy in the 

competing partials surrounding the partial being 
followed. Again a noise floor must be established to 
prevent the adjustment of partials that are actually 
just .low-level noises, as stated in the previous 
3 0 methods . 

In all threshold and adjustment methods, the 
thresholds (single threshold or separate upper and 
lower thresholds) may not be flat, because the human 
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ear itself is not flat. The ear does not recognize 
amplitude in a uniform or linear fashion across the 
audible range* Instead, higher frequencies must have 
more energy added to them than lower frequencies to 
obtain a given perceived increase in loudness. 
Because our hearing response is frequency- dependent 
(some frequencies are perceived to have greater 
energy than others) , the adjustment of energy in the 
present invention is also frequency-dependent* 

By interpolating the adjustment amount between 
a maximum and minimum amplitude adjustment, a more 
continuous and consistent adjustment can be 
achieved- For example, a partial with an amplitude 
near the maximum level (near clipping) would be 
adjusted downward in energy more than a partial 
whose amplitude was barely exceeding the downward- 
adjustment threshold. Time thresholds are set so 
competing partial s in a set frequency range have 
limits. Threshold curves and adjustment curves may 
represent a combination of user-desired definitions 
and empirical perceptual curves based on human 
hearing . 

Figure 9A shows a sample threshold curve and 
Figure 9B an associated sample adjustment curve for 
threshold and adjustment method 1. The thresholds 
are dependent upon the overall signal energy (e.g., 
a lower overall energy would lower the thresholds) . 
When an incoming partial 's amplitude exceeds the 
upper energy threshold curve, or ceiling of Figure 
9A, the partial is cut (adjusted downward) in energy 
by an amount defined by the associated adjustment 
curve for that frequency of Figure 9B. Likewise, 
when a partial 1 s amplitude drops below the lower 
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energy threshold curve, or floor, its energy is 
boosted (adjusted upward) , once again by an amount 
defined by the associated adjustment curve for that 
frequency. The increase and/ or reduction in 
amplitude may be by some predetermined amount. 

The adjustment curves of Figure 9B define the 
maximum amount of adjustment made at a given 
frequency. To avoid introducing distortion into the 
partial 1 s amplitude, the amount of adjustment is 
tapered in time, such that there is a smooth 
transition up to the maximum adjustment. The 
transition period may be defined by an arbitrary 
function, and may be as simple as a linear pattern. 
Without a gradual taper, a waveform may be adjusted 
too quickly, or create discontinuities, which create 
undesirable and/or unwanted distortions in the 
adjusted signal. Similarly, tapering is also 
applied when adjusting the partial upward. 

Figure 9C shows an example that relates to the 
second threshold and adjustment method. 

Over the duration of a signal, its 
harmonics/partials may be fairly constant in 
amplitude, or they may vary, sometimes considerably, 
in amplitude. These aspects are frequency and time 
dependent, with the amplitude and decay 
characteristics of certain harmonics behaving in one 
fashion in regard to competing partials. 

Aside from the previously discussed thresholds 
for controlling maximum amplitude and minimum 
amplitude of harmonics (either as individual 
harmonics or as groups of harmonics), there are also 
tixne-based thresholds which may be set by the user. 
These must be met in order for the present invention 
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to proceed with its adjustment of partials. 

Time-based thresholds set the start time, 
duration, and finish time for a specified 
adjustment, such that amplitude thresholds must be 
5 met for a time period specified by the user in order 

for the present invention to come into play. If an 
amplitude threshold is exceeded, for example, but 
does not remain exceeded for the time specified by 
the user, the amplitude adjustment is not processeid. 

10 For example, a signal falling below a minimum 

threshold either (1) once met that threshold and 
then fell below it; or (2) never met it in the first 
place also are not adjusted. It is useful for the 
software to recognize such differences when 

15 adjusting signals and be user adjustable. 

INTERPOLATION 
In general terms, interpolation is a method of 
estimating or calculating an unknown quantity in 

20 between two given quantities, based on the 

relationships among the given quantities and known 
variables. In the present invention, interpolation 
is applicable to Harmonic Adjustment, Harmonic 
Adjustment and Synthesis, Partial Transformation, 

25 and Harmonic Transformation- This refers to a 

method by which the user may adjust the harmonic 
structure of notes at certain points sounded either 
by an instrument or a human voice. The shift in 
harmonic structure all across the musical range from 

3 0 one of those user-adjusted points to the other is 

then affected by the invention according to any of 
several curves or contours or interpolation 
functions prescribed by the user. Thus the changing 
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harmonic content of played notes is controlled in a 
continuous manner. 

The sound of a voice or a musical instrument 
may change as a function of register. Because of the 
varying desirability of sounds in different 
registers , singers or musicians may wish to maintain 
the character or timbre of one register while 
sounding notes in a different register. In the 
present invention, interpolation not only enables 
them to do so but also to adjust automatically the 
harmonic structures of notes all across the musical 
spectrum from one user-adjusted point to another in 
a controllable fashion. 

Suppose the user desires an emphasis on the 3rd 
harmonic in a high-register note, but an emphasis on 
the 10th harmonic in the middle register. Once the 
user has set those parameters as desired, the 
present invention automatically effects a shift in 
the harmonic structure of notes in between those 
points, with the character of the transformation 
controllable by the user. 

Simply stated, the user sets harmonics at 

certain points, and interpolation automatically 

adjusts everything in between these "set points." 

More specifically, it accomplishes two things : 

• First, the user may adjust the harmonic 
structure of a note (or group of notes 
within a selected range) of a voice or 
instrument at different points within that 
voice or instrument's range; in doing so, 
the user may be correcting perceived 
deficiencies in the sound, or adjusting 
the sound to produce special effects, or 
emphasizing harmonics deemed desirable, or 
diminishing or deleting harmonics deemed 
undesirable, or whatever the case may be; 



• Second, once the. user has adjusted the 
sounds of these selected notes or 
registers, the present invention shifts or 
transforms the harmonic structure of all 
notes and all perceived harmonics all 
across the musical spectrum in between the 
set points, according to a formula pre- 
selected by the user. 

The interpolation function (that is, the 
character or curve of the shift from one set point's 
harmonic structure to another) may be linear, or 
logarithmic, or of another contour selected by the 
user. 

A frequency scale can chart the location of 
various notes, harmonics, partials, or other 
signals. For example, a scale might chart the 
location of frequencies an octave apart. The manner 
in which the present invention adjusts all harmonic 
structures between the user's set points may be 
selected by the user: the transformation may be 
linear, logarithmic, following a fragment of a conic 
section, or whatever the user selects. 

IMITATING NATURAL HARMONICS 
A good model of harmonic frequencies is f n » n x 
f x x S Xoa 2 n because it can be set to approximate 
natural "sharping" in broad resonance bands. For 
example, the 10 th harmonic of f 1 = 185 Hz is 1862.3 
Hz instead of 1850 Hz using 10 x 185. More 
importantly, it is the one model which simulates 
consonant harmonics, e.g., harmonic 1 with harmonic 
2, 2 with 4, 3 with 4, 4 with 5, 4 with 8, S with 8, 
8 with 10, 9 with 12, etc. When used to generate 
harmonics those harmonics will reinforce and ring 
even more than natural harmonics do. It can also be 
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used for harmonic adjustment and synthesis, and 
natural harmonics. This function or model is a good 
way of finding closely matched harmonics that are 
produced by instruments that ^sharp" higher 
5 harmonics. In this way, the stretch function can be 

used in Imitating Natural Harmonics INH. 

The function f a - f x x n x (S) l0 * 2 a is used to 
model harmonics which are progressively sharper as n 
increases. S is a sharping constant, typically set 

10 between 1 and 1.003 and n is a positive integer 1, 

2, 3, w , T, where T is typically equal to 17. With 
this function, the value of S determines the extent 
of that sharping. The harmonics it models are 
consonant in the same way harmonics are consonant 

15 when f n « n x f x . I.e., if f n and f m are the n 01 and 

m th harmonics of a note, then 

f J f tn= t f 2 n/ f 2m s=s f3n/f3m=- =3 fkn/fkm 

There are multitudes of methods that can be 
utilized to determine the location of fundamental 

20 and harmonic f requencies, such as Fast -Find 

Fundamental, Short -Time Fourier Transform methods, 
or the explicit locating of frequencies through 
filter banks or auto-correlation techniques. The 
degree of accuracy and speed needed in a particular 

25 operation is user-defined, which helps aid in 

selecting the appropriate frequency- finding 
algorithm. 

SEPARATING HARMONICS FOR EFFECTS 
30 A further extension of the present invention 

and its methods allows for unique manipulations of 
audio, and application of the present invention to 
other areas of audio processing. Harmonics of 
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interest are selected by the user and then separated 
from the original data by the use of previously 
mentioned variable digital filters. Filtering 
methods used to separate the signal may be of any 
method, but particularly applicable are digital 
filters whose coefficients may be recalculated based 
on input data. 

The separated harmonic (s) are then fed to other 
signal processing units (e.g., effects for 
instruments such as reverberation , chorus, flange, 
etc.) and finally mixed back into the original 
signal in a user-selected blend or proportion. 

IMPLEMENTATION 

One implementation variant includes a source of 
audio signals 22 connected to a host computer 
system, such as a desktop personal computer 24, 
which has several add-in cards installed into the 
system to perform additional functions. The source 
32 may be live or from a stored file. These cards 
include Analog- to-Digital Conversion 26 and Digital - 
to-Analog Conversion 28 cards, as well as an 
additional Digital Signal Processing card that is 
used to carry out the mathematical and filtering 
operations at a high speed. The host computer 
system controls mostly the user- interface 
operations. However, the general personal computer 
processor may carry out all of the mathematical 
operations alone without a Digital Signal Processor 
card installed. 

The incoming audio signal is applied to an 
Analog-to-Digital conversion unit 26 that converts 
the electrical sound signal into a digital 
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representation. In typical applications, the 
Analog-to-Digital conversion would be performed 
using a 20 to 24 -bit converter and would operate at 
48kHz - 96kHz [and possibly higher] sample rates. 
Personal computers typically have 16-bit converters 
supporting 8kHz - 44.1kHz sample rates. These may 
suffice for some applications. However, large word 
sizes - e.g., 20 bits, 24 bits, 32 bits - provide 
better results. Higher sample rates also improve 
the quality of the converted signal . The digital 
representation is a long stream of numbers that are 
then stored to hard disk 30. The hard disk may be 
either a stand-alone disk drive, such as a high- 
performance removable disk type media, or it may be 
the same disk where other data and programs for the 
computer reside. For performance and flexibility, 
the disk is a removable type. 

Once the digitized audio data is stored on the 
disk 30, a program is selected to perform the 
desired manipulations of the signal. The program 
may actually comprise a series of programs that 
accomplish the desired goal. This processing 
algorithm reads the computer data from the disk 32 
in variable- sized units that are stored in Random 
Access Memory (RAM) controlled by the processing 
algorithm. Processed data is stored back to the 
computer disk 30 as processing is completed. 

In the present invention, the process of 
reading from and writing to the disk may be 
iterative and/or recursive, such that reading and 
writing may be intermixed, and data sections may be 
read and written to many times. Real-time 
processing of audio signals often requires that disk 
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accessing and storing of the digital audio signals 
be minimized, as it introduces delays into the 
system. By utilizing RAM only, or by utilizing 
cache memories, system performance can be increased 
to the point where some processing may be able to be 
performed in a real-time or quasi real-time manner. 
Real-time means that processing occurs at a rate 
such that the results are obtained with little or no 
noticeable latency by the user. Dependent upon the 
processing type and user preferences, the processed 
data may overwrite or be mixed with the original 
data. It also may or may not be written to a new 
file altogether. 

Upon completion of processing, the data is read 
from the computer disk or memory 30 once again for 
listening or further external processing 34. The 
digitized data is read from the disk 30 and written 
to a Digital -to-Analog conversion unit 28, which 
converts the digitized data back to an analog signal 
for use outside the computer 34 . Alternately, 
digitized data may written out to external devices 
directly in digital form through a variety of means 
(such as AES/EBU or SPDIF # digital audio interface 
formats or alternate forms) . External devices 
include recording systems, mastering devices, audio- 
processing units, broadcast units, computers, etc. 

Processing occurs at a rate such that the 
results are obtained with little or no noticeable 
latency by the user. Dependent upon the processing 
type and user preferences, the processed data may 
overwrite or be mixed with the original data. It 
also may or may not be written to a new file 
altogether. 
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Upon completion of processing, the data is read 
from the computer disk or memory 30 once again for 
listening or further external processing 34. The 
digitized data is read from the disk 30 and written 
5 to a Digital -to -Analog conversion unit 28 , which 

converts the digitized data back to an analog signal 
for use outside the computer 34. Alternately, 
digitized data may written out to external devices 
directly in digital form through a variety of means 
10 (such as AES/EBU or SPDIF digital audio interface 

formats or alternate forms) . External devices 
include recording systems, mastering devices, audio 
processing units, broadcast units, computers , etc. 

15 Fast Find Harmonics 

The implementations described herein may also 
utilize technology such as Fast -Find Fundamental 
Method to process in quasi real time. This Fast- 
Find Method technology uses algorithms to deduce the 

20 fundamental frequency of an audio signal from the 

harmonic relationship of higher harmonics in a very 
quick fashion such that subsequent algorithms that 
are required to perform in real-time may do so 
without a noticeable (or with an insignificant) 

25 latency. The Fast -Find algorithm may provide 

information as to the location of harmonic 
frequencies such that processing of harmonics may be 
carried out fast and efficiently. 

The method includes selecting at least two 

>0 candidate frequencies in the signal. Next, it is 

determined if the candidate frequencies are a group 
of legitimate harmonic frequencies having a harmonic 
relationship. Finally, the fundamental frequency is 
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deduced from the legitimate frequencies. 

In one method, relationships between and among 
detected partial s are compared to comparable 
relationships that would prevail if all members were 
legitimate harmonic frequencies . The relationships 
compared include frequency ratios, differences in 
frequencies, ratios of those differences, and unique 
relationships which result from the fact that 
harmonic frequencies are modeled by a function of 
harmonic ranking number. Candidate frequencies are 
also screened using the lower and higher limits of 
the fundamental frequencies and/or higher harmonic 
frequencies which can be produced by the source of 
the signal. 

The method uses relationships between and among 
higher harmonics, the conditions which limit 
choices, the relationships the higher harmonics have 
with the f undamental , and the range of possible 
fundamental frequencies. f n = f x x n x G(n) models 
the frequency of the nth harmonic . Examples are : 

a) Ratios of candidate frequencies f H , f M , f L/ 
must be approximately equal to ratios obtained by 
substituting their ranking numbers R H , R M , R L in the 
model of harmonics, i.e., f H +■ f M « {R H x G (R H ) } *{R M 
X G (Rm) } , and f M * f L «{R M x G (R„)}*{R L x G (R L ) } . 

b) The ratios of differences between candidate 
frequencies must be consistent with ratios of 
differences of modeled frequencies, i.e., 

(R„ - R M )- 5 -(R M • RJ - [{R„ x G(R„) }-{R M x G(R M ) }]-,[{r m 
x G(R M ) }±{R lt x G(R L ) }] . 

c) The candidate frequency partials f H , f Mf f L 
must be in the range of frequencies which can be 
produced by the source or the instrument. 
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d) The harmonic ranking numbers R H , R M , R L must 
not imply a fundamental frequency which is below F L 
or above F H , the range of fundamental frequencies 
which can be produced by the source or instrument. 
5 e) When matching integer variable ratios to 

obtain possible trios of ranking numbers, the 
integer R„ in the integer ratio R H /R M must be the 
same as the integer R„ in the integer ratio Rm/Rj., 
for example. This relationship is used to join 
10 Ranking Number pairs {R H , R„ }and {R„, }into 

possible trios {r h , R„, R^ } . 

The candidate frequency and its ranking number 
can be used in the previously described methods even 
with out deducing the fundamental frequency to 
15 modify or synthesize harmonics of interest . 

Another method for determining legitimate 
harmonic frequencies and deducing a fundamental 
frequency includes comparing the group of candidate 
frequencies to a fundamental frequency and its 
20 harmonics to find an acceptable match. This 

includes, creating a harmonic multiplier scale for 
the fundamental and all of its harmonics. A 
candidate partial frequency scale is created with 
the candidate frequencies and compared to the 
25 harmonic multiplier scale to find an acceptable 
match. The ranking number of the candidate 
frequencies is determined from the match of the two 
scales. These ranking numbers are then used to 
determine whether the group is a group of legitimate 
30 frequencies. If this is so, the match can also be 

used to determine the fundamental frequency or 
further calculation can be performed. Preferably, 
the scales are logarithmic scales. 
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The present invention does not rely solely on 
Fast -Find Fundamental to perform its operations* 
There are multitudes of methods that can be utilized 
to determine the location of fundamental and 
5 harmonic frequencies , such as Short-Time Fourier 

Transform methods, or the explicit locating of 
frequencies through filter banks or auto -correlation 
techniques. The degree of accuracy and speed needed 
in a particular operation is user-defined, which 

10 helps aid in selecting the appropriate frequency- 

finding algorithm. 

The potential inter- relationship of the various 
systems and methods for modifying complex waveforms 
according to the principles of the present invention 

15 are illustrated in Figure 11. Input signals 

provided to a sound file as complex waveforms. This 
information can then be provided to a Fast Find 
Fundamental method or circuitry. This may be used 
to quickly determine the fundamental frequency of a 

20 complex waveform or as a precursor to provide 

information for further Harmonic Adjustment and/or 
Synthesis. This is especially true if the analysis 
is to be done quasi -real time. 

Harmonic Adjustment and/or Synthesis is based 

25 on a moving target or modifying devices being 

adjustable with respect to amplitude and frequency. 
In an offline mode, the Harmonic 
Adjustment/Synthesis would receive its input 
directly from the sound file. The output can be 

30 just from Harmonic Adjustment /Synthesis . 

Alternatively, Harmonic Adjustment Synthesis 
signal in combination with any of the separating 
Harmonics for Effects, Interpolation or Imitating 
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Natural Harmonics may be provided as an output 
signal . 

Harmonic Actuation based on moving targets may 
also receive an input signal off-line directly from 
5 the input of the sound file of complex waveforms or 

as an output form the Harmonic Adjustment and/or 
Synthesis. It provides an output signal either out 
of the system or as a input to Harmonic 
Transformation. The Harmonic Transformation is 

10 based as well as on moving target and includes 

target files, interpolation and imitating natural 
harmonics . . 

The present invention has been described in 
words such that the description is illustrative of 

15 the matter. The description is intended to describe 

the present invention rather than in a manner of 
limitation. Many modifications, combinations , and 
variations are possible of the methods provided 
above. It should therefore be understood that the 

20 invention may be practiced in ways other than 

specifically described herein. 
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WHAT IS CLAIMED 

1. A method of manipulating a complex 
waveform which includes harmonic signals having a 
fundamental and higher harmonic frequencies, the 
method comprising: 

determining ranks of harmonic frequencies of 
harmonic signals present in the waveform; 
modifying preselected ranks of the determined 
harmonic frequencies to manipulate the waveform; 
and 

repeating the steps of determining and 
modifying to modify the same preselected ranks of 
harmonic frequencies with changes of the frequencies 
of the harmonic signals in the waveform* 

2. A method according to Claim 1, wherein one 
or more of a) the preselected ranks and b) the 
modifications varies with time within the waveform- 

3. A method according to Claim 1, wherein 
modifying includes adjusting the amplitude of the 
preselected ranks of harmonic frequencies to a 
predetermined value- 

4- A method according to Claim 1, wherein 
modifying includes adjusting the amplitude of the 
preselected ranks of harmonic frequencies relative 
to partial frequencies in the waveform. 

5. The method according to Claim 1, wherein 
modifying includes modifying, by rank, the waveform 
from one source to resemble a signal of a second 
source . 
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6. A method according to Claim 1, wherein 
modifying includes determining missing preselected 
ranks of harmonic frequencies, synthesizing the 
missing harmonic frequencies and adding the 
synthesized harmonic frequencies to the waveform. 

7. The method according to Claim 6, wherein 
the missing harmonic frequencies are synthesized 
using a modeling function (S) 1o * 2 \ where S is a 
constant greater than 1 and n is the rank of the 
harmonic . 

8. The method according to Claim 1, wherein 
modifying includes determining missing preselected 
inharmonic frequencies, synthesizing the missing 
inharmonic frequencies and adding the synthesized 
inharmonic frequencies to the waveform. 

9. The method according to Claim 1, wherein 
determining the rank includes: selecting frequencies 
in the waveform; determining which of the selected 
frequencies have a harmonic relationship and the 
harmonic relationship; and determining the 
frequencies of the preselected rank from the 
selected frequencies determined to have a harmonic 
relationship . 

10. The method according to Claim 9 , wherein 
determining the harmonic relationship includes using 
a modeling function (s) where S is a constant 
greater than 1 and n is the harmonic rank. 

11. The method according to Claim 1, including 
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a plurality of modifying means with adjustable with 
respect to at least one of frequency and amplitude 
and including processing the harmonic frequencies of 
the preselected ranks by the modifying means in 
which its frequency lies. 

12. The method according to Claim 11, 
including assigning a harmonic rank to each 
modifying means and tuning the modifying means to 
the frequency of the determined harmonic 
frequencies . 

13. The method according to Claim 11, wherein 
the modifying means are set to a fixed frequencies; 
and including adjusting the modifying effect of the 
modifying means as a function of the frequency of 
the preselected harmonic frequencies. 

14. A method according to Claim 1, including 
storing the method as instructions in a digital 
signal processor. 

15. A method of modifying the harmonic content 
of a complex waveform comprising: 

determining the harmonic frequencies of the 
waveform; 

determining the presence of preselected ranks 
of harmonic frequencies in the waveform; 

synthesizing the preselected harmonic 
frequencies absent from the waveform; 

adding any synthesized harmonic frequencies to 
the waveform; and, 

repeating the steps of determining and 
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synthesizing to modify the same preselected ranks of 
harmonic frequencies with changes of the frequency 
of the harmonic frequencies in the waveform. 

16. The method according to Claim 15, wherein 
the missing harmonic frequencies are synthesized 
using a modeling function (S) where S is a 
constant greater than 1 and n is the rank of the 
harmonic. 

17. The method according to Claim 15 , 
including adjusting the amplitude of preselected 
ranks of harmonic frequencies present in the 
waveform. 

18. The method according to Claim 17, 
including a plurality of modifying means with 
adjustable with respect to at least one of frequency 
and amplitude and including processing the harmonic 
frequencies of the preselected ranks by the 
modifying means in which its frequency lies. 

19. The method according to Claim 15, 
including storing the method as instructions in a 
digital signal processor. 

20. A method of harmonic adjustment comprising: 
storing a complex waveform; 

determining harmonics in the waveform including 
their frequency, amplitude and harmonic rank over 
time; 

adjusting the amplitudes of selected harmonics 
based on harmonic rank and preset parameters. 
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21. The method according to Claim 20, 
including determining harmonics using a modeling 
function IS) l09 2 n , where S is a constant greater than 
1 and n is the harmonic rank. 

22. The method according to Claim 20, 
including comparing harmonics to adjacent partial 
frequencies in the waveform and adjusting harmonics 
relative to the harmonics based on the conqparisons. 

23. The method according to Claim 20, 
including setting two or more frequency based 
parameters; selecting an interpolation function; and 
adjusting the amplitudes of harmonics wave based on 
the frequency based parameters and interpolation 
function. 

24. The method according to Claim 20, 
including synthesizing selected harmonics based on 
harmonic rank and preset parameters and adding any 
synthesized harmonics to the waveform. 

25. The method according to Claim 20, 
including a plurality of modifying means with 
adjustable with respect to at least one of frequency 
and amplitude and including processing the harmonic 
frequencies of the preselected ranks by the 
modifying means in which its frequency lies. 

26. The method according to Claim 20, 
including storing the method as instructions in a 
digital signal processor. 
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27. A method of harmonic adjustment comprising: 

determining harmonics including their 
frequency , amplitude and harmonic rank over time of 
a first sound waveform; 

determining harmonics including their 
frequency, amplitude and harmonic rank over time of 
a target sound waveform; 

adjusting the amplitudes of harmonics of the 
first sound waveform based on the amplitude of 
corresponding rank harmonics of the target sound 
waveform* 

28. The method according to Claim 27, 
including synthesizing harmonics and inharmonics 
missing in the first sound waveform compared to the 
target waveform base on harmonic rank. 

29. The method according to Claim 27, 
including a plurality of modifying means with 
adjustable with respect to at least one of frequency 
and amplitude and including processing the harmonic 
frequencies of the preselected ranks by the 
modifying means in which its frequency lies. 

30. The method according to Claim 27, 
including storing the method as instructions in a 
digital signal processor. 

31. A method of manipulating of a complex 
waveform comprising: 

determining partial frequencies and their 
amplitudes in the waveform over time; 

comparing the amplitude of adjacent partial 
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frequencies; and 

adjusting the amplitude of selected partial 
frequencies based on the comparison. 

32. A method according to Claim 31, wherein 
the comparing and adjustment moves with the partial 
frequencies over time. 

33. A method according to Claim 31 , including 
adjusting the amplitude to maintain the selected 
partial frequencies within between a pair of 
thresholds. 

34. A method of manipulating of a complex 
waveform comprising: 

calculating an amplitude threshold dynamic over 
time; and 

adjusting the amplitude of the partials to 
converse to the dynamic threshold over time. 

35. A method according to Claim 34 , including 
determining the degree of adjustment based on the 
amount the partials deviate from the threshold. 

36. A method according to Claim 34, including 
determining a noise threshold for the complex 
waveform and not adjusting the partials having an 
amplitude less than the noise floor. 
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AMENDED CLAIMS 

[received by the International Bureau on 25 April 2000 (25.04.00); 
original claims 1-4, 7-10-13, 15, 16, 1 8, 21-23, 25, 27, 29 and 34 amended: 
original claim 35 cancelled; original claim 36 renumbered as claim 35; new claims 36-44 added: 

remaining claims unchanged (9 pages)] 
1. A method of modifying a complex waveform 
which includes periodic complex cones having 
harmonic frequencies, the method comprising: 

determining ranks of harmonic frequencies of 
complex tones present in the waveform; 

modifying preselected ranks of the determined 
harmonic frequencies to modify the waveform; and 

repeating the steps of determining and 
modifying to modify the same preselected ranks of 
harmonic frequencies with changes of the frequencies 
of the harmonic signals in the waveform. 

2. The method according to Claim 1, wherein 
the modifying of the preselected ranks varies with 
time within the waveform. 

3. The method according to Claim 1, wherein 
modifying includes adjusting the amplitude of the 
preselected ranks of harmonic frequencies by a 
predetermined value. 

4. The method according to Claim 1, wherein 
modifying includes adjusting the amplitude of the 
preselected ranks of harmonic frequencies relative 
to the amplitudes of other preselected ranks in the 
waveform. 

5. The method according to Claim 1, wherein 
modifying includes modifying, by rank, the waveform 
from one source to resemble a signal of a second 
source . 
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6. The method according to Claim 1, wherein 
modifying includes determining missing preselected 
ranks of harmonic frequencies, synthesizing the 
missing harmonic frequencies and adding the 
synthesized harmonic frequencies to the waveform. 

7. The method according to Claim 6, wherein 
the missing harmonic frequencies are synthesized 
using a modeling function n x <S) l09 2 n , where S is a 
constant greater than 1 and n is the rank of the 
harmonic. 

8. The method according to Claim 1, wherein 
modifying includes determining missing preselected 
noninteger inharmonics, synthesizing the missing 
inharmonic s and adding the synthesized inharmonics 
to the waveform. 

9. The method according to Claim 1, including 
determining which partials are harmonics of the same 
harmonic tone spectrum and their harmonic ranks 
using the methods of Past Find Fundamental. 

10. The method according to Claim 1, including 
setting two or more frequency based parameters; 
selecting an interpolation function; and adjusting 
the amplitudes of harmonics based on the frequency 
based parameters and interpolation function. 

11. The method according to Claim 1, including 
a plurality of modifying means adjustable with 
respect to at least one of frequency and amplitudes; 
and including processing the harmonic frequencies of 
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the preselected ranks by the modifying means in 
which the harmonic frequency lies. 

12. The method according to Claim 11, 
including assigning a harmonic rank to each 
modifying means and tuning the modifying means to 
the frequency of the determined harmonic frequency 
of that rank. 

13. The method according to Claim 11, wherein 
the modifying means are set to a fixed frequencies; 
and including adjusting the modifying effect of the 
modifying means as a function of the preselected 
rank of the harmonic frequencies. 

14. The method according to Claim 1, including 
storing the method as instructions in a digital 
signal processor. 

15. A method of modifying the harmonic content 
of a complex waveform comprising: 

determining the harmonic frequencies of the 
waveform; 

determining the presence of preselected ranks 
of harmonic frequencies in the waveform; 

synthesizing the preselected harmonic 
frequencies absent from the waveform; 

adding any synthesized harmonic frequencies to 
the waveform; and 

repeating the steps of determining and 
synthesizing to modify the same preselected ranks of 
harmonic frequencies with changes of the frequency 
of the harmonic frequencies in the waveform. 
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16. The method according to Claim 15 , wherein 
the missing harmonic frequencies are synthesized 
using a modeling function n x (S l09 2 n ) * where S is a 
constant greater than 1 and n is the rank of the 
harmonic . 

17. The method according to Claim 15 , 
including adjusting the amplitude of preselected 
ranks of harmonic frequencies present in the 
waveform, 

18. The method according to Claim 17 , 
including a plurality of modifying means with 
adjustable respect to at least one of frequency and 
amplitude; and including processing the harmonic 
frequencies of the preselected ranks by the 
modifying means in which the harmonic frequency 
lies, 

19 . . The method according to Claim 15 , 
including storing the method as instructions in a 
digital signal processor. 

20. A method of harmonic adjustment comprising: 
storing a complex waveform; 

determining harmonics in the waveform including 
their frequency, amplitude and harmonic rank over 
time; 

adjusting the amplitudes of selected harmonics 
based on harmonic rank and preset parameters. 

21. The method according to Claim 20, wherein 
the adjusting is accomplished using frequency and 
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amplitude adjustable digital filters, 

22. The method according to Claim 20 , 
including comparing harmonics to adjacent harmonics 
in the waveform and adjusting the harmonics relative 
to the adjacent harmonics based on the comparisons. 

23. The method according to Claim 20, 
including setting two or more frequency based 
parameters; selecting an interpolation function; and 
adjusting the amplitudes of harmonics based on the 
frequency based parameters and interpolation 
function. 

24. The method according to Claim 20 , 
including synthesizing selected harmonics based on 
harmonic rank and preset parameters and adding any 
synthesized harmonics to the waveform. 

25. The method according to Claim 20, 
including a plurality of modifying means adjustable 
in both frequency and amplitude; and including 
processing the harmonic frequencies of the 
preselected ranks by the modifying means. 

26. The method according to Claim 20, 
including storing the method as instructions in a 
digital signal processor. 

27. A method of modify harmonic content of a 
complex waveform comprising: 

determining harmonics including their 
frequency, amplitude and harmonic rank over time of 
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a first sound waveform;. 

determining harmonics including their 
frequency, amplitude and harmonic rank over time of 
a target sound waveform; 

adjusting the amplitudes of harmonics of the 
first sound waveform by ranking number based on the 
amplitude of corresponding rank harmonics of the 
target sound waveform over time. 

28* The method according to Claim 27 , 
including synthesizing harmonics and inharmonics 
missing in the first sound waveform compared to the 
target waveform base on harmonic rank. 

29. The method according to Claim 27, 
including a plurality of modifying means adjustable 
with respect to at least one of frequency and 
amplitude; and including processing the harmonic 
frequencies of the preselected ranks by the 
modifying means in which the harmonic frequency 
lies. 

30. The method according to Claim 27 , 
including storing the method as instructions in a 
digital signal processor. 

31. A method of manipulating of a complex 
waveform comprising: 

determining partial frequencies and their 
amplitudes in the waveform over time; 

comparing the amplitude of adjacent partial 
frequencies ; and 

adjusting the amplitude of selected partial 
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frequencies based on the comparison. 

32. The method according to Claim 31, wherein 
the comparing and adjustment moves with the partial 
frequencies over time. 

33. The method according to Claim 31 , 
including adjusting the amplitude to maintain the 
selected partial frequencies within between a pair 
of thresholds, 

34 . A method of modifying a complex waveform 
comprising: 

calculating an amplitude threshold dynamic over 

time; 

determining the degree of adjustment based on 
the amount partials deviate from the threshold; and 

adjusting the amplitude of the partials by the 
determine degree of adjustment 

35. The method according to Claim 34, 
including determining a noise threshold for the 
complex waveform and not adjusting the partials 
having an amplitude less than the noise floor. 

36. A method of modifying the amplitudes of 
harmonics of known ranks in a complex waveform, the 
method comprising: 

modifying the amplitudes of pre- selected 
harmonics by ranking number; and 

repeating the modifying steps as the 
frequencies of the various pre-selected harmonic 
ranks change in time. 
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37. A method of modifying the amplitudes of 
harmonics of known ranks in a complex waveform, the 
method comprising: 

modifying the amplitudes of pre-selected 
harmonics by ranking number; 

synthesizing pre-selected harmonic ranks; 

adding synthesized harmonic ranks to the 
waveform; and 

repeating the steps as the waveform changes , in 

time. 

38. A method of modifying the amplitudes of 
harmonics in a complex waveform based on the 
amplitudes in a target waveform, the method 
comprising: 

determining the differences in relative 
amplitudes of the harmonics by frequency range in 
the attack, sustain and decay phases of each note; 

modifying the amplitudes of harmonics by rank; 

synthesizing missing harmonics by frequency 
range in the attack, sustain and decay phases of 
each note, and adding them to the waveform; and 

repeating the steps as the waveform changes in 

time. 

39. A method of modifying the amplitudes of 
partials in a complex waveform, the method 
comprising: 

setting an average energy threshold variable 
with frequency and dynamically adjustable over time; 

setting a noise floor threshold varying with 
frequency; 

determining the partial' s amplitudes in 
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relation to the thresholds; 

modifying the amplitudes of the partials as a 
function of the determined relation of the partial ' s 
amplitude to the thresholds and a scaling function; 
and 

repeating the steps as the partials change in 

time. 

40. The method according to one of Claims 36 
to 38 , including determining which partials are 
harmonics of the same harmonic tone spectrum using 
the methods of Past Find Fundamental. 

41. The method according to one of Claims 36 
to 39, wherein modifying is accomplished using 
frequency & amplitude adjustable digital filtering. 

42. The method according to one of Claims 36 
to 38 including determining harmonic ranks using the 
methods of Fast Find Fundamental. 

43. The method according to Claim 38 wherein 
adjusting amplitudes of harmonics includes adjusting 
amplitudes of partials and inharmonic it ies by 
frequency range in the attack, sustain, and decay 
phases of each note. 

44. The method according to Claim 39 including 
setting an upper energy threshold that varies with 
frequency. 
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